Diagnostic system having gaze tracking

ABSTRACT

A method of modifying a portion of a digital image. In one arrangement, a digital image is displayed on a display, and the gaze data of a user viewing the image on the display is determined. The portion of the image being viewed by the user corresponding to the determined gaze data to identify a gaze region is then determined. The identified gaze region is modified to generate a modified gaze region, and the resulting modified gaze region is displayed on the display.

FIELD OF THE INVENTION

The present invention relates to imaging systems and, more particularly,to imaging systems enabling interactions with digital images based ongaze data, that is, information related to the movement/tracking of ahuman eye.

BACKGROUND OF THE INVENTION

Systems enabling interactions with digital images based on gaze data areknown. For example, European Patent Application No. EP1335270 describesa method and system for using non-manual commands, such as voice andgaze, to control the actions of a medical image digital display system.Patent Application No. EP1335270 describes a method for using non-manualcommands, such as voice and gaze, to control the actions of a medicalimage digital display system. Images are provided to the medical imagedigital display system to be displayed and controlled by a computersystem using non-manual commands that do not require the use of thelimbs of the hands or feet.

U.S. patent application Ser. No. 2004/0183749 relates to a method andapparatus for improving communications between humans and devices bymeans of providing an attentive user interface for obtaining informationabout an attentive state of a user and modulating operation of a deviceon the basis of the obtained information. The information about theuser's attentive state is obtained through eye contact of the user withthe device that is sensed by the attentive user interface.

U.S. patent application Ser. No. 2004/0212712 relates to a digitalimaging device which detects or tracks a user's direction of gaze. Thedetected direction of gaze may be used to set one or more functions inthe digital imaging device, and the history of the user's direction ofgaze may be stored with a captured digital image for later use by anautomatic image post-processing apparatus.

U.S. Pat. No. 6,152,563 is directed to a system for eye-gaze directiondetection that uses an infrared light emitting diode mounted coaxiallywith the optical axis and in front of the imaging lens of an infraredsensitive video camera for remotely recording images of the eye of thecomputer operator. When coupled with a computer screen and a graphicaluser interface, the system may place the cursor at the user's point ofregard and then perform the various mouse clicking actions at thelocation on the screen where the user fixates.

U.S. Pat. No. 6,243,095 relates to a navigation and display systemhaving a cursor on the graphical user interface, which is coupled to amouse, where movement of the mouse controls the location of the cursorwith respect to the icons. The icons are arranged such that handmovements required to control the mouse correspond to head and eyemovements made by radiologists when viewing similar images on hard copyfilm.

While such systems may have achieved certain degrees of success in theirparticular applications, there exists a need for a system that enablesviewing and interacting with regions of digital images, based on gazedata.

SUMMARY OF THE INVENTION

An object of the present invention is to provide a system that can beused to promote viewing of digital images for medical diagnosticpurposes.

Another object of the present invention is to provide a system that canbe used to promote viewing of digital images for airport securitypurposes.

Another object of the present invention is to provide a system that canbe used to promote viewing of digital images for quality inspectionpurposes

Another object of the present invention is to provide a system that canbe used to promote interacting with digital images for entertainmentpurposes

These objects are given only by way of illustrative example, and suchobjects may be exemplary of one or more embodiments of the invention.Other desirable objectives and advantages inherently achieved by thedisclosed invention may occur or become apparent to those skilled in theart. The invention is defined by the appended claims.

According to one aspect of the invention, there is provided a method ofmodifying a portion of a digital image. The method include the steps of:displaying the digital image on a display; determining gaze data of auser viewing the image displayed on the display; determining the portionof the image being viewed by the user corresponding to the determinedgaze data to identify a gaze region; modifying the identified gazeregion to generate a modified gaze region; and displaying the modifiedgaze region on the display.

According to another aspect of the invention, there is provided a methodof annotating a portion of a digital image. The method includes thesteps of: displaying the digital image on a display; determining gazedata of a user viewing the image displayed on the display; determiningthe portion of the image being viewed by the user corresponding to thedetermined gaze data to identify a gaze region; providing annotationdata for the identified gaze region; and associating the annotation datawith the identified gaze region.

According to yet another aspect of the invention, there is provided amethod for comparing image portions of two images wherein methodincludes the steps of: displaying a first digital image on a display;determining gaze data of a user viewing the first digital imagedisplayed on the display; determining the portion of the first digitalimage being viewed by the user corresponding to the determined gaze datato identify a first gaze region; accessing a second digital image;determining, in the second digital image, a second gaze regioncorresponding to the first gaze region; and displaying the first andsecond gaze regions on the display.

BRIEF DESCRIPTION OF THE DRAWINGS

The foregoing and other objects, features, and advantages of theinvention will be apparent from the following more particulardescription of the embodiments of the invention, as illustrated in theaccompanying drawings.

The elements of the drawings are not necessarily to scale relative toeach other.

FIG. 1 is a schematic of an imaging system in accordance with thepresent invention.

FIG. 2 is a flow diagram showing a method of local image enhancementbased on gaze data in accordance with the present invention.

FIG. 3 is a flow diagram showing a method of local voice annotationbased on gaze data in accordance with the present invention.

FIG. 4 is a flow diagram showing a method of local image comparisonbased on gaze data in accordance with the present invention.

FIG. 5 is a flow diagram illustrating a method utilizing gazeinformation to provide local CAD data in accordance with the presentinvention.

DETAILED DESCRIPTION OF THE INVENTION

The following is a detailed description of the preferred embodiments ofthe invention, reference being made to the drawings in which the samereference numerals identify the same elements of structure in each ofthe several figures.

FIG. 1 shows a schematic of an imaging system 10 in accordance with anembodiment of the present invention for use by a user 12. System 10includes a computer or computer station 14 in communication with adisplay monitor/unit 16 viewable by user 12. Display unit 16 can utilizevarious known display technologies, for example, CRT, LCD, OLED, plasma,projection, and the like.

A typical user of system 10 is a radiologist or other healthcarepractitioner. However, system 10 can be employed by other users needingimage analysis and viewing, for example, scientists, engineers, securitypersonal, designers, and the like.

Computer station 14 is in communication with a gaze tracking device 18,that is a device which can monitor/track/determine the movement/focus ofat least one eye of user 12.

Gaze tracking device 18 can be a standalone device such as, for example,the Tobii x50 system manufactured by Tobii Company, Sweden.Alternatively, gaze tracking device 18 can be integral with display unit16 such as, for example, Tobii 1750 system manufactured by TobiiCompany, Sweden. In yet another alternative arrangement, gaze trackingdevice 18 can be a head-mounted device wearable by user 12 such as, forexample, the ASL 501 gaze tracking system manufactured by AppliedScience Laboratories, MA.

Gaze tracking device 18 is in communication with computer station 14using means known to those skilled in the art. Such means can include acable employing a common interface, such as the well known UniversalSerial Bus (USB) interface or the IEEE 1394 interface. Alternatively,communication/connection can be established using a wireless interface,such as the well known Bluetooth interface or the IEEE Standard 802.15interface.

System 10 further includes one or more input devices, such as a computermouse 20, keyboard 22, or a microphone 24. Other suitable input devicescan include a trackball, computer pad, joystick, touch-screen and thelike.

Input devices 20,22,24 are connected to computerstation 14 by meansknown to those skilled in the art, for example, a cables employing acommon interface, such as the well known Universal Serial Bus (USB)interface or the IEEE 1394 interface. Alternatively, the connection canbe established using a wireless interface, such as the well knownBluetooth interface or the IEEE Standard 802.15 interface.

Computer station 14 includes a hard drive storage device 26 adapted tostore digital data such as digital images, digitized x-rays, CT scans,MRI images, and the like. Alternatively, the digital data can bedownloaded to computer station 14 from a remote server 28 using througha communication cable/network 30.

Computer station 14 is provided with appropriate application softwarefor tracking and utilizing gaze data of user 12 in accordance with thepresent invention. The application software can be stored on storagedevice 26 or on removable media device (such as a CD-ROM disc) readableby computer station 14. Alternatively, the application software can beaccessed/operated/downloaded from remote server 28.

System 10 can be employed for various applications. According to oneembodiment of the present invention, system 10 is used formodifying/enhancing a local region of a digital image wherein theaforementioned local region is determined by the gaze of user 12.

An objective of modifying/enhancing a digital medical image is toimprove the visibility and/or acuity of the observed phenomena to thehuman observer so as to increase the diagnostic performance of the user.Often, monitors/displays do not have enough dynamic range to display adigital medical image on the monitor/display without clipping/losingpart of the image. Medical images can have dynamic ranges in excess of12 bits whereas the typically display monitor has only display 8 bits ofdata. Therefore, a rendering function is often used to convert themedical image data into a range that is suitable to be displayed on themonitor.

One mapping function linearly scales the high dynamic range medicalimage such that it fits within the dynamic range of the monitor. Thescaling preserves the contrast but the displayed image can suffer from aloss of visibility of detail due to quantization. In addition, therendering functions can also be used to enhance the contrast of imagestructures allowing abnormalities to more readily visible to theobserver. The dilemma exists as to how to display a high dynamic rangemedical image on a low dynamic range display while preserving andenhancing the visual detail the human observer needs in order to make anaccurate diagnosis. In some cases, the observer is only interested inexamining a small portion of the image at a time. Therefore, it ispossible to more efficiently use the bit depth by modifying/enhancingonly the local region of interest instead of trying to modify/enhancethe entire image.

One method of determining the region of interest (ROI, also referred toas a local region or local gaze region) of a displayed image is totrack/monitor an observer's gaze (i.e., eye movement and/or focus) anddetermine the observer's current fixation position. Then, theenhancement is applied only to the region of interest that is determinedby the gaze of the observer. The local gaze region acts as a spatialwindowing function in that only the data within the local gaze region isused to modify/enhance the image. The enhanced local image can thendisplayed on the monitor, and if desired, it can be displayed on themonitor superimposed on the original image.

The local gaze region is defined as a sub-region of the displayedmedical image around the current gaze position of the observer. For atwo dimensional medical image such as a projection radiograph, the localgaze region corresponds to a circular (or alternatively, rectangular)shaped region around the gaze point/position of the observer. For athree-dimensional image, the local gaze region corresponds to aspherical (or alternatively, rectangular box) shaped region around thegaze point/position of the observer. The extent (e.g., radius or lineardimension) of the local gaze region can be a function of one or morefactors, for example: 1) the size of the original medical image; 2) thesize of the screen window in which the image is displayed; 3) the sizeof the display/monitor and/or 4) a predetermined value defined by theuser.

Referring to FIG. 2, there is shown a flow diagram illustrating oneembodiment of a method in accordance with the present inventionemploying system 10 wherein gaze information is used to locally enhancea digital image.

Initially, a digital image is accessed and displayed on display 16,shown in FIG. 2 as acquisition and display step 102.

In a gaze data acquisition step (shown as step 104), the currentlocation of the observers gaze is obtained from gaze tracking device 18.User 12's eye is constantly moving, saccadic motion, about anobject/region of interest and is unable to fixate on a pointvoluntarily. The saccadic motion leads to a rapid 2-dimensionalBrownian-like motion around and between fixation points. In a region ofinterest determination step 106, the gaze data is filtered to remove thesaccadic motion so as to determine the local gaze region.

Known methods can be employed to remove the saccadic motion, forexample, it can either be removed by averaging or median filtering ofthe last n gaze data or by using more sophisticated fixationidentification algorithms such as described in D. D. Salvucci and J. H.Goldberg, “Identifying Fixations and Saccades in Eye-Tracking Protocol”,Proceedings of the Eye Tracking Research and Application Symposium, pp.71-88, New York, ACM Press.

In an enhancement/modification step 108, the image (i.e., the imagecontent/data) within the local gaze region is extracted and processedthrough a modification/enhancement algorithm to modify/increase thevisually contrast between adjacent structures. Histogram modificationtechniques can be employed, such as histogram equalization or contrastlimited histogram equalization as described in S. M. Pizer and E. P.Auburn, “Adaptive histogram equalization and its variations”, ComputerVision, Graphics, and Image Processing, vol. 39, pp. 355-368, 1987, toenhance the image within the local gaze region. Alternatively,multi-scale techniques that enhance the image adaptively based upon thespatial frequency properties can be used to enhance the image within thelocal gaze region. An example of a multi-scale enhancement technique isdescribed in Y. Jin, L. Fayad, and A. Laine, “Contrast Enhancement byMulti-scale Adaptive Histogram Equalization,” in Wavelet Applications inSignal and Image Processing IX, Proceedings of SPIE, San Diego Calif.,4478, pp. 206-213, July 2001.

In a show enhanced region step 110, the enhanced image (resulting fromstep 108) is superimposed upon the original image in the location fromwhich it was extracted. Alternatively, the enhanced image can beenlarged and superimposed upon the original image, giving the appearanceof the original image being viewed by an image enhancing magnifyingglass. An optional border can be drawn around the enhanced image to helpdelineate the enhanced image from the original image and/or theunenhanced region of interest.

Referring now to step 112, the method of the present invention canrepeat steps 104-110 so as to enhance the local gaze region until user12 indicates (in decision step 112) that the enhancement of the localgaze region is no longer desired, wherein the process stops (step 114).

In addition to, or alternatively, the method of the present inventioncan repeat steps 108-110 so as to repeatedly modify/enhance the localgaze region until user 12 indicates (in decision step 112) that theenhancement of the local gaze region is no longer desired, wherein theprocess stops (step 114). The same local gaze region might bemodified/enhanced in differing/various ways.

According to another method in accordance with the present invention,system 10 can be used for annotating a local region of an image whereinthe local region of interest is determined by the gaze of the user.

The ability to annotate a local region of interest can assist the userin relating what he/she sees with what he/she says. Using this feature,for example, a radiologist can distinctly/precisely indicate anabnormality with little or no description of where the abnormality islocated. A link between the abnormal region and radiologist's report canbe stored automatically, and later readily retrieved for subsequentusage.

Typically, radiologist reports are shared with other doctors such asreferring physicians, surgeons, specialists, and the like. By means ofestablishing an automatic linkage between a radiologist's dictation andthe region of interest, diagnostic findings can be ready shared betweenmedical doctors thereby reducing the chance of miscommunication and/orinterpretation errors.

Referring now to FIG. 3 there is shown a flow diagram illustrating amethod utilizing gaze information to locally annotate a digital image.Steps 202 through 206 mirror steps 102 through 106 of FIG. 2, and so arenot further described.

In an annotation step 208, an annotation related to a region of interestis obtained using one of the input devices, such as computer mouse 20,keyboard 22, microphone 24 or other suitable input device. In oneembodiment, the annotation is acquired from microphone 24 and is storedas a voice file by computer station 14. In another embodiment, theannotation is acquired from keyboard 20 or mouse 20 and stored as a textfile by computer station 14.

It is recognized that the annotation can be any information—textual orvisual or audible—desired to be associated with the local gaze region.For example, the annotation can be an symbol, video, notes, results oflocal image analysis, results of computer aided detection system, eyetracking history, eye tracking history for each of the viewers or forsuccessive viewers.

At step 210, computer station 14 creates an annotation tag/markerindicating a name of the annotation file and corresponding region ofinterests determined from the gaze data. The term “metadata” is usuallyused to describe any information related to an image. Examples of othertypes of image metadata include image capture time, capture device,image capture parameters, image editing history, and the like.Alternatively, the information about the annotation file andcorresponding region of interest can be stored in a separate file inassociation with the image identifier.

System 10 can repeat steps 204 through 210 to enable the annotation ofthe local gaze region until user 12 indicates, at step 212, that furtherannotation is no longer desired, wherein the process stops at step 214.

In addition to, or alternatively, the method of the present inventioncan repeat steps 208 and 210 so as to repeatedly annotate the local gazeregion until user 12 indicates (in decision step 212) that sufficientannotation has been provided to the local gaze region, wherein theprocess stops (step 214). The same local gaze region might be annotatedin differing/various ways.

System 10 can be configured to provide a visual indicator on the displayor on the digital image indicating the association of the annotationdata with the identified gaze region. For example, a unique symbol mightbe displayed such that user 12 (or subsequent viewers) can detect theexistence of an annotation.

If the digital image has been previously annotated, system 10 canprovide an audible indicator to user 12 when the digital image is firstdisplayed (at steps 102 in FIG. 2 and step 202 in FIG. 3).

In addition, system 10 can provide an audible indicator indicating, atcompletion of the association (step 210), the successful association ofannotation data with the identified gaze region.

According to another embodiment of the present invention, the systemdepicted in FIG. 1 can be used for comparing regions of interest in atleast two images. The images can represent different projections on ofthe same part of the body, or they can be obtained in different timeperiods and/or imaging methods.

Referring now to FIG. 4 there is shown a flow diagram illustrating amethod utilizing gaze information to locally compare digital images.

Steps 302 through 306 for a first image i₁ mirror steps 102 through 106of FIG. 2, and so are not further described.

In step 308, a spatial alignment of the first image i₁ and the secondimage i₂ is established. There is a number of ways to establish spatialalignment between digital images. In one embodiment, the images areobtained by the same device (e.g., Computer Tomography (CT) scan) andthey represent two different projections (i.e., two CT slices). In thisembodiment, the coordinate systems for both images match each other.

In another embodiment, the images are obtained by different devices andthey represent two different modalities (e.g., a Computer Tomography(CT) slice and an Ultrasound (US) image). In this embodiment, theimaging coordinate systems need to be calibrated to each other in orderhave spatial alignment.

In yet another embodiment, the spatial alignment for multimodal imagesrelies on artificial objects attached to the patient, object with aredesigned to be well visible and accurately detectable in all of therelevant modalities.

In yet still another embodiment, the spatial alignment relies onanalyzing image content. In this embodiment, the alignment is based on alimited set of identified salient points (landmarks), on the position ofsegmented binary structures, most commonly object surfaces, or directlyonto measures computed from the image gray values.

These methods are well known to those skilled in the art in the medicalimage analysis field, such methods being described in literature, forexample, in J. B. A. Maintz and M. A. Viergever, “A survey of medicalimage registration,” Medical Image Analysis 2(1), pp. 1-36, 1998.

In yet another embodiment, the spatial alignment for digital images isestablished based on gaze data obtained for reference points. In thisembodiment, the user looks at a region in the first image i₁, then at acorresponding region in the second image i₂, and the mean value of dazedata for the two regions are used to align the two images.

A combination of spatial alignment methods is also possible. Forexample, the user looks a region in the first image i₁, then at acorresponding region in the second image i₂. Consequently, imageanalysis is performed in both gazed regions, salient features in theseregions are determined, and finally the spatial alignment is establishedbased on the alignment of the corresponding salient features.

In step 310, the region of interest for a second image i₂ isdefined/determined based on the spatial alignment completed in step 308.

In step 312, the regions of interest for both the first i₁ and second i₂images are displayed for an observer's comparison. In one embodiment,the corresponding regions of interest are displayed in two separatewindows side-by-side. If the region of interest for the first image ismoved as a result of the gaze change, the corresponding region ofinterest for the second image can be automatically moved as well.Additionally, both images can be repositioned automatically inside theirwindows such that the corresponding regions of interests are alwaysdisplayed in the center of the windows. The repositioning can be alsoinitiated using various input devices, such as keyboard 20, computermouse 22, and/or microphone 24.

In another embodiment, the user can toggle between two regions ofinterest, which are shown in the same window using either verbal ornon-verbal commands.

In yet another embodiment, the corresponding regions of interest overlayeach other, so that the user could see the difference between them. Thiscan be accomplished by making one region of interest semi-transparent.

In yet another embodiment, the user can look at a difference imageobtained as a result of subtraction of the region of interest of thesecond image from the region of interest of the first image.Additionally, other image processing and or display methods can beutilized that compare corresponding regions of interest.

Image comparison can be done with more than two images. For example, inone embodiment, the corresponding regions of interest are displayed inthree separate windows for three CT slices (one large on the top of thedisplay and two small on the bottom). If the region of interest for thefirst image (on the bottom) is moved as a result of the gaze change, thecorresponding regions of interest for the second and third images (onthe bottom) can be automatically moved as well. Additionally, all imagescan be repositioned automatically inside their windows such that thecorresponding regions of interests are always displayed in the center ofthe windows.

System 10 can repeat steps 304 through 312 to enable the localcomparison of images until user 12 indicates, at step 314, that furthercomparison is no longer desired, wherein the process stops at step 316.

According to another method in accordance with the present invention,system 10 can be used for demonstrating results of computer aideddetection (CAD) method for a local region of an image wherein the localregion of interest is determined by the gaze of the user.

Referring now to FIG. 5 there is shown a flow diagram illustrating amethod utilizing gaze information to provide local CAD (computer aideddetection) data. Steps 402 through 406 mirror steps 102 through 106 ofFIG. 2, and so are not further described.

In step 408, results of a CAD method for the gazed region of interestare obtained. In another embodiment, results of a CAD method for thewhole image are obtained. CAD methods are known. For example, PatentApplication No. WO/2004029851, filed Sep. 24, 2003 by Menhardt, Zhang,and Harrington describes a CAD method for abnormalities detection inX-ray images in which the features used in the CAD analysis aredisplayed as coded descriptors to provide an indication to a user of aparticular abnormality therefore increasing the reliability of adiagnosis established by the user.

In step 410, CAD results are shown to the user based on the gaze data.In one embodiment, the CAD data i shown for the gazed region ofinterest. In this embodiment, the user may want to look only for CADdata within the gazed region of interest and not be distracted by otherCAD data.

In another embodiment, the CAD data for all other regions except for thegazed data are shown. This embodiment is useful if the user wants tocheck whether he/she miss any suspicious area with possibleabnormalities, i.e., area without gaze data.

At step 412, system 10 can repeat steps 404 through 410 to enable thelocal view of CAD data until user 12 indicates that further inspectionis no longer desired, wherein the process stops at step 416.

The invention has been described in detail with particular reference toa presently preferred embodiment, but it will be understood thatvariations and modifications can be effected within the spirit and scopeof the invention. The presently disclosed embodiments are thereforeconsidered in all respects to be illustrative and not restrictive. Thescope of the invention is indicated by the appended claims, and allchanges that come within the meaning and range of equivalents thereofare intended to be embraced therein.

1. A method of annotating a portion of a digital image, comprising thesteps of: (a) displaying the digital image on a display; (b) determininggaze data of a user viewing the image displayed on the display; (c)determining the portion of the image being viewed by the usercorresponding to the determined gaze data to identify a gaze region; (d)receiving annotation data for the identified gaze region from the user;(e) associating the annotation data with the identified gaze region; (f)displaying the identified gaze region; and (g) displaying a visualindicator on the display or on the digital image indicating theassociation of the annotation data with the identified gaze region. 2.The method of claim 1, further comprising the step of providing anaudible indicator indicating the association of annotation data with theidentified gaze region.
 3. The method of claim 1, wherein the steps of(b) through (e) are repeated.
 4. The method of claim 1, wherein thesteps of (d) and (e) are repeated.
 5. The method of claim 1, furthercomprising the steps of: modifying the identified gaze region togenerate a modified gaze region; and displaying the modified gaze regionon the display.